Metadata recovery for de-duplicated data

ABSTRACT

A data stream is stored in storage media. As part of the storage, the data stream is divided into a plurality of chunks. The plurality of chunks include a target chunk that is next to a first chunk in a file within the data stream. A determination is made that the target chunk matches an existing chunk stored in the storage media. In response to the determination, a first pointer to the existing stored chunk is created in file metadata for the file. Also in response to the determination, a second pointer to a first stored chunk that matches the first chunk is created in chunk metadata embedded with the existing stored chunk.

BACKGROUND

The present disclosure relates generally to the field of dataprocessing, and, more particularly, to metadata recovery forde-duplicated data.

As the amount of information being stored continues to increase everyyear, the importance of intelligently managing data storage has becomemore important. One technique for managing data storage isde-duplication. This technique is used in many computing environments todecrease the amount of space required to store a given quantity of data.

SUMMARY

Embodiments of the present disclosure include a method for storing adata stream in storage media. As part of the method, the data stream isdivided into a plurality of chunks. The plurality of chunks include atarget chunk that is next to a first chunk in a file within the datastream. A determination is made that the target chunk matches anexisting chunk stored in the storage media. In response to thedetermination, a first pointer to the existing stored chunk is createdin file metadata for the file. Also in response to the determination, asecond pointer to a first stored chunk that matches the first chunk iscreated in chunk metadata embedded with the existing stored chunk.

Embodiments of the present disclosure further include a computer programproduct for managing a file stored on storage media. The file includesfile data that is stored as a plurality of chunks on a first storageentity of the storage media. Each stored chunk includes chunk metadataembedded therewith. The file further includes file metadata that isstored on a second storage entity of the storage media. The computerprogram product is a computer readable storage medium that has programinstructions embodied thereon. The program instructions are configuredto cause a computer to perform a method. As part of the method, loss orcorruption of the stored file metadata is detected. In response to thedetection, a recovery operation is performed. As part of the recoveryoperation, chunk metadata embedded with a first stored chunk of theplurality of chunks is read. Based on reading the chunk metadataembedded with the first stored chunk, a determination is made that thefirst chunk does not have any preceding chunks in the file. Also basedon reading the chunk metadata embedded with the first stored chunk, apointer to a second stored chunk is identified. Based on the pointer, adetermination is made that the second chunk follows the first chunk inthe file. A recovered version of the file metadata is written in thesecond storage entity. Based on the determination that the first chunkdoes not have any preceding chunks in the file and further based on thedetermination that the second chunk follows the first chunk in the file,the recovered version of the file metadata indicates that the firstchunk is the initial chunk of the file and the second chunk follows theinitial chunk in the file.

Embodiments of the present disclosure further include a system. Thesystem includes a processor and a memory. The processor is incommunication with the memory and is configured to obtain instructionsfrom the memory that cause the processor to perform a method. As part ofthe method, a data stream including a file to be stored in storage mediais received. The storage media includes a data storage entity andmetadata storage entity. The received file is divided into a pluralityof chunks. Each chunk of the plurality of chunks is compared withexisting chunks stored in the data storage entity. For each chunk of theplurality of chunks that does not match any of the existing chunks, thechunk is stored in the data storage entity. If the stored chunk is notthe first chunk in the file or the last chunk in the file, a metadatafield that includes a pointer to a chunk following the stored chunk inthe file is embedded with the stored chunk. If the stored chunk is thefirst chunk in the file, a metadata field that includes a pointer to achunk following the stored chunk in the file and an indicator that thestored chunk is the first chunk in the file is embedded with the storedchunk. If the stored chunk is the last chunk in the file, a metadatafield that includes an indicator that the stored chunk is the last chunkin the file is embedded with the stored chunk. File metadata stored inthe metadata storage entity is updated to include a pointer to thestored chunk. For each chunk of the plurality of chunks that does matchan existing chunk, the chunk is not-stored in the data storage entity.If the not-stored chunk is not the first chunk in the file or the lastchunk in the file, a metadata field embedded in the existing chunk isupdated to include a pointer to a chunk following the not-stored chunkin the file. If the not-stored chunk is the first chunk in the file, themetadata field embedded in the existing chunk is updated to include apointer to a chunk following the not-stored chunk in the file and anindicator that the not-stored chunk is the first chunk in the file. Ifthe not-stored chunk is the last chunk in the file, the metadata fieldembedded in the existing chunk is updated to include an indicator thatthe not-stored chunk is the last chunk in the file. File metadata storedin the metadata storage entity is updated to include a pointer to theexisting chunk.

The above summary is not intended to describe each illustratedembodiment or every implementation of the present disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

The drawings included in the present disclosure are incorporated into,and form part of, the specification. They illustrate embodiments of thepresent disclosure and, along with the description, serve to explain theprinciples of the disclosure. The drawings are only illustrative oftypical embodiments and do not limit the disclosure.

FIG. 1 illustrates a flow diagram of a method for storing de-duplicateddata on storage media, in accordance with embodiments of the presentdisclosure.

FIG. 2 illustrates a block diagram of an example file of a data streambeing de-duplicated for storage in storage media, in accordance withembodiments of the present disclosure.

FIG. 3 illustrates a block diagram of data and metadata being stored aspart of de-duplication of an example set of files, in accordance withembodiments of the present disclosure.

FIG. 4 illustrates a block diagram of an example data chunk withembedded metadata, in accordance with embodiments of the presentdisclosure.

FIG. 5 illustrates a block diagram of a data structure and a separatemetadata structure being used to store a set of de-duplicated files, inaccordance with embodiments of the present disclosure.

FIG. 6 illustrates a flow diagram of a method for obtaining files from ade-duplicated data storage environment, in accordance with embodimentsof the present disclosure.

FIG. 7 illustrates a flow diagram of a method of recovering filemetadata within a de-duplicated data storage environment, in accordancewith embodiments of the present disclosure.

FIG. 8 illustrates a block diagram of an example recovery history table,in accordance with embodiments of the present disclosure.

FIG. 9 illustrates a high-level block diagram of an example computersystem that may be used in implementing embodiments of the presentdisclosure.

While the embodiments described herein are amenable to variousmodifications and alternative forms, specifics thereof have been shownby way of example in the drawings and will be described in detail. Itshould be understood, however, that the particular embodiments describedare not to be taken in a limiting sense. On the contrary, the intentionis to cover all modifications, equivalents, and alternatives fallingwithin the spirit and scope of the invention.

DETAILED DESCRIPTION

Aspects of the present disclosure relate generally to the field of dataprocessing, and, more particularly, to metadata recovery forde-duplicated data. While the present disclosure is not necessarilylimited to such applications, various aspects of the disclosure may beappreciated through a discussion of various examples using this context.

In some embodiments, the storage of de-duplicated files involves storingfile data separately from file metadata for the purpose of savingstorage space. If the file metadata is lost or corrupted, however, theresult may be that the file data itself becomes unusable, and the entireset of files can only be recovered by re-copying them from a backupsource. This is often a serious burden on computing resources. In someembodiments, this unfortunate situation can be avoided by embeddingcertain metadata with the file data itself. Then, in the event that themain file metadata is lost, it can be recovered using the metadata thatis embedded with the file data. This can result in large savings ofcomputing resources when file metadata recovery becomes necessary.

Referring now to the figures, shown in FIG. 1 is a flow diagram of amethod 100 for storing de-duplicated data on storage media, inaccordance with embodiments of the present disclosure. In someembodiments, operations of the method 100 may be performed by aprocessor of a computer storage system. The method 100 may begin atoperation 101, wherein the system begins receiving a data stream. Thestream may be received, for example, from a remote computer attemptingto store, on the storage media, a set of files included in the stream.Per operation 102, the data stream is divided (e.g., split) into aplurality of chunks (e.g., data fragments) in a process referred to aschunking. These chunks may be of fixed or variable size depending on thestorage parameters of the system.

Per operation 103, an individual chunk is identified in the data stream.Per operation 104, the identity characteristics of the selected chunkare determined. The identity characteristics may include aspects of theselected chunk that distinguish it from other chunks. In someembodiments, determining the identity characteristics of a chunkinvolves creating a hash of all or a portion of the chunk. Per operation104, a pattern search is performed. This pattern search may involvecomparing the identity characteristics of the selected chunk to identitycharacteristics of other existing chunks already stored in the storagemedia. For example, this may involve comparing the hash of the selectedchunk to hashes of the existing chunks. These hashes of existing chunksmay be stored in a memory index that is updated with a new hash eachtime a corresponding new chunk is stored in the storage media.

Per operation 106, a determination is made, based on the identitycharacteristic comparisons, as to whether the selected chunk matches anyexisting (e.g., previously stored) chunk. If the selected chunk does notmatch any existing chunk, then the method proceeds to operation 107,wherein the selected chunk is stored as a new chunk in the storagemedia. Per operation 108, a new pointer (e.g., reference, locationidentifier) is created for the new chunk. This pointer may be stored ina metadata structure that hosts the file metadata for the file of whichthe selected chunk is a part. Per operation 109, a new counter for thenew chunk is created (e.g., written). In some embodiments, a counter mayserve to indicate the number of times that a particular chunk isincluded in files stored in the storage media. As new filesincorporating that chunk are added to the storage media the counter isincreased and when old files incorporating that chunk are removed (e.g.,deleted) from the storage media the counter is decreased. In someembodiments, the counter may aid garbage collection activities in thestorage media (e.g., when the counter hits “0”, the corresponding chunkcan be erased from storage).

Per operation 110, a record is added to new chunk metadata (e.g., ametadata field) that is embedded with (e.g., appended to) the new chunkin the storage media. This record includes a pointer to the chunkpreceding (e.g., immediately in front of) the selected chunk in the fileand a second pointer to the chunk following (e.g., immediately after)the selected chunk in the file. Additional metadata may also be includedin the record, including, for example, an identifier of the file towhich the selected chunk belongs. Per operation 111, the memory index isupdated to include the identity characteristics of the new chunk so thatother chunks may analyzed against that information as part of futurepattern match searches.

Returning now to operation 106, if the selected chunk does match anexisting chunk, then the method proceeds to operation 112 (rather than107), wherein a new pointer to the existing chunk is created. Thispointer may be stored in a metadata structure that hosts the filemetadata for the file of which the selected chunk is a part. Peroperation 113, a counter for the existing chunk is updated (e.g.,increased by 1). Per operation 114, a record is added to existing chunkmetadata that is embedded with the existing chunk in the storage media.This record includes a pointer to the chunk preceding the selected chunkin the file and a second pointer to the chunk following the selectedchunk in the file.

Upon completion of operation 111 or 114, the method proceeds tooperation 115, wherein a determination is made as to whether there areadditional chunks in the data stream that need to be stored. If thereare additional chunks, then the method loops back to operation 103, andoperations 103 to 115 are repeated, as applicable. If there are noadditional chunks in the data stream, then the method ends at operation116.

Referring now to FIG. 2, shown is a block diagram of an example file 202of a data stream 201 being de-duplicated for storage in storage media231, in accordance with embodiments of the present disclosure. The blockdiagram includes embodiments of the method 100 being performed on thefile 202. As shown, the data stream 201 is received by a storage system.The data stream 201 includes a file 202 of user data. The series ofcombined data portions making up the file 202 is represented in thediagram by the series of segments with non-identical segments numbereddifferently. This file 202 is sent to a chunker 203 that divides thefile 202 into its constituent chunks 211-218. As shown, the chunks arethen stored as chunks 221-223 in storage media 231. As stored, thechunks 221-223 each include embedded metadata (represented by the linedportions of each stored chunk).

As the chunks 211-218 are de-duplicated, the new chunks that match(e.g., are identical to) existing chunks are not stored, but rather anew pointer to the corresponding existing chunk is added to the filemetadata (not shown) for the file 202 and additional pointers (topreceding and following chunks) are added to the chunk metadata embeddedwith the corresponding existing chunk. For new chunks that do not matchexisting chunks, the chunk is stored, a new pointer to the newly storedchunk is added to the file metadata, and the additional pointer areadded to the newly stored chunk's embedded metadata. In someembodiments, the stored chunks 221-223 are compressed to further savespace in the storage media 231.

In an example relevant to FIG. 2, the file 202 is received and chunk 211is selected. A pattern search reveals that there are no existing chunksin the storage (e.g., no chunks stored as part of preceding files in thedata stream 201 or chunks received in prior data streams). Chunk 211 istherefore stored as chunk 221 and the file metadata for file 202 andchunk metadata for new chunk 221 are created/updated. Later, when chunks216 and 218 are processed, they are not stored as chunks in storagemedia 231, instead the file metadata for file 202 and the chunk metadatafor the existing (previously stored) chunk 221 are created/updated.

Referring now to FIG. 3, shown is a block diagram of data and metadatabeing stored as part of de-duplication of an example set of files301-303, in accordance with embodiments of the present disclosure. Asshown, the data and metadata of example files A, B, and C (301-303) areseparated for storage of the files. In some embodiments, file data andfile metadata are stored in separate storage entities. As used herein, astorage entity may refer to a logically and/or physically distinctstorage area. A storage entity may include, for example, a disk, apartition within a disk, or a logical unit number (LUN).

As shown, the combined file metadata 311 includes the file identifier(file ID), and a sequence of pointers to the ordered chunks for each ofthe files 301-303. For example, the file metadata for File A includesits file ID and a pointer to a stored chunk matching its first chunkfollowed by a pointer to a stored chunk matching its second chunkfollowed by a pointer to a stored chunk matching its third chunk. Thisfile metadata 311 is to be stored in a specific metadata structurewithin a storage entity.

The data of each file is divided into chunks 321-324 which are to bestored, in a non-duplicated manner, in a specific data storage structurewithin a storage entity. Embedded with each chunk 321-324 iscorresponding chunk metadata 331-334. In the depicted example, eachchunk metadata 331-334 includes a metadata field, with each row of thefield representing a separate record. These records may be stored inchronological order within each chunk metadata field (e.g., with theoldest (e.g., first processed) record being listed first and the newestrecord being listed last).

In the depicted example, each record includes metadata about a separateinstance of the corresponding chunk (or, more precisely, a chunkmatching the corresponding chunk) being included in a file to be stored.Specifically, each record includes the file ID of the file that includedthe relevant chunk, a pointer to a stored chunk matching the precedingchunk in that file, and a pointer to a stored chunk matching thefollowing chunk in that file. If a chunk is the first (e.g., initial)chunk in a file, then the record corresponding to that chunk includes anindicator of this primacy. For example, as depicted, the indicator caninclude a null value in the portion of the metadata field used forlisting a preceding chunk. Likewise, if a chunk is the last chunk in afile, then a corresponding indicator can include a null value in theportion of the metadata field used for listing a following chunk.

In an example relevant to FIG. 3, a file 302 (File B) is received aspart of a data stream. The file 302 includes a sequence of user data ofchunk 2, chunk 1, chunk 1, and chunk 4. Upon storage of file 302 instorage media, the file metadata 311 for the file 302 includes a fileID, a pointer to stored chunk 322, followed by a pointer to stored chunk321, followed by a second pointer to stored chunk 321, followed by apointer to stored chunk 324. The storage of file 302 in the storagemedia also results in the chunk metadata 332, 331, and 334 beingupdated. Specifically, a record is created in chunk metadata 332 thatincludes a File ID of B, a null preceding chunk value, and a pointer tostored chunk 321 for the following chunk value. Furthermore, two recordsare created in chunk metadata 331 that include a File ID of B, pointersto stored chunks 322 and 321 for preceding chunk values, and pointers tostored chunks 321 and 324 for following chunk values. Finally, anadditional record is created in chunk metadata 334 that includes a FileID of B, a pointer to stored chunk 321 for the preceding chunk value,and a null following chunk value.

Referring now to FIG. 4, shown is a block diagram of an example datachunk 401 with embedded metadata 402, in accordance with embodiments ofthe present disclosure. As shown, chunk 401 (user data chunk N) hasembedded metadata 402 that includes two additional categories of chunkmetadata that are not depicted in the embedded metadata shown in FIG. 3.Specifically, in the example of embedded metadata 402 each recordincludes a file length for the identified file and a unique recordidentifier (record ID) for that record. In some embodiments, including arecord ID for each record aids in identifying (e.g., marking) records asthey are recovered during an operation to recover file metadata (e.g.,method 700 described in reference to FIG. 7 herein). In otherembodiments, other recovery indicator, such as a recovery bit, may beincluded with each record to indicate whether or not it has beenrecovered yet as part of a file metadata recovery operation.

Likewise, including a file length identifier in records can also aid infile metadata recovery operations. For example, upon recovery of a givenfile, this length information may be checked to confirm that therecovered file is the correct length. If the length is incorrect, thenit is known that file was not properly recovered.

Referring now to FIG. 5, shown is a block diagram of a data structure501 and a separate metadata structure 502 being used to store a set ofde-duplicated files, in accordance with embodiments of the presentdisclosure. As shown, data structure 501 is located on a first LUN (LUN1). The LUN 1 is organized as group of several storage units 511-513.Each storage unit 511-513 includes a plurality of rows of data storageblocks. For example, storage unit 511 includes a first row 521, a secondrow 522, and an nth row 523. The blocks in the rows may all be the samesize or may have variable sizes. The data structure 501 is used to storefile data. The chunks of the stored file data may be stored sequentiallyor randomly within LUN 1. Each chunk is stored with embedded metadata(represented in the figure by the lines the chunks) that includespointers to preceding and following chunks within individual files.

The metadata structure 502 is located on a second LUN (LUN 2). Themetadata structure 502 is used to store file metadata that correspondsto file data stored on the data structure 501. In the depicted example,the file metadata is arranged by file IDs 541-543 for all of then filesstored on data structure 501. Each file ID 541-543 in the metadatastructure 502 is linked with a set of pointers to the chunks of theidentified file in the data structure 501. These pointers aresequentially ordered for each File ID 541-543.

As shown, two files (File A and File B) are stored in this storagemedia. File A includes a data sequence of chunk 3, chunk 2, chunk 5,chunk 5, and chunk 1. File B includes a sequence of chunk 4, chunk 1,and chunk 3. These chunks are stored on the data structure 501 (withembedded metadata) as stored chunks 531-535. The file metadata for bothfiles is stored on the metadata structure 502. For example, the filemetadata for File A includes the file ID 541 for that file which isstored with sequential pointers to the stored chunks of that file,namely, a first pointer to stored chunk 531 followed by a second pointerto stored chunk 534 followed by a third pointer to stored chunk 533followed by a fourth pointer to stored chunk 533 followed by a fifthpointer to stored chunk 535. In this example, each storage block in thedata structure is 512 kb, and each pointer includes a storage unitnumber, row number, and offset value to identify the location of thecorresponding stored block. For example, the pointer to stored chunk 535indicates that the storage unit number is 1, that the row number is 2,and that the offset is 1536 kb. From this information, the location ofstored chunk 535 is easily determined.

Referring now to FIG. 6, shown is a flow diagram of a method 600 forobtaining files from a de-duplicated data storage environment, inaccordance with embodiments of the present disclosure. In someembodiments, operations of the method 600 may be performed by aprocessor of a computer storage system. The method 600 may begin atoperation 601, wherein the system receives a request for a file storedin storage media. The request may be received from, for example, aremote computer that needs to view the file. Per operation 602, the filemetadata for the requested file is located in a metadata structurewithin the storage media. Per operation 603, sequential pointers withinthe file metadata are used to identify the locations of the chunks ofdata that make up the file. Per operation 604, the contents of the datachunks, which are located in a data structure within the storage media,are read to put together the file. Per operation 605, the file isprovided to the requester (e.g., the remote computer). Per operation606, the method ends. It should be noted that, in some embodiments ofthe method 600, the embedded metadata stored with each chunk are notused (e.g., are ignored) for the purposes providing requested filesduring the course of normal operations of the system. Instead, thepointers stored in the metadata structure are used to gather thefragments of the requested file. This may have significant operationalbenefits (in terms of time and computing resources) as compared to usingthe chunk metadata when obtaining files.

An example use of operations of the method 600 is now described withreference to the structures of FIG. 5. In this example, a user on aremote computer requests to access a copy of File A. The request is sentto a storage system that manages storage media upon which File A isstored. The system locates the file ID 541 for File A in the metadatastructure 502. Using the pointers located with File ID 541, the systemreads out stored chunk 531, stored chunk 534, stored chunk 533, storedchunk 533, and stored chunk 535 from the data structure 501. The systemthen combines the chunks to reconstitute File A. The embedded metadatain each of the chunks may or may not be included in the reconstitutedFile A. The system then provides File A to the remote computer.

Referring now to FIG. 7, shown is a flow diagram of a method 700 ofrecovering file metadata within a de-duplicated data storageenvironment, in accordance with embodiments of the present disclosure.In some embodiments, operations of the method 700 may be performed by aprocessor of a computer storage system. Specifically, the system mayperform the method 700 in response to detecting that file metadata forone, some, or all of the files stored on storage media has been lost orcorrupted. For example, method 700 in may be performed in response todetermining that the metadata structure 502 of FIG. 5 has beencorrupted.

The method may begin at operation 701, wherein a chunk stored in a datastorage structure (e.g., chunk 531 of data structure 501 of FIG. 5) isselected. Per operation 702, metadata embedded with the stored chunk isread. Per operation 703, a determination is made as to whether there isany unrecovered record in the chunk metadata that has a null value forits preceding chunk pointer. If not, then the method loops back tooperation 701 and new chunk (e.g., the next chunk in data structure) isselected. If there are unrecovered records with null preceding chunkpointer values, then, per operation 704, the oldest unrecovered recordin that chunk metadata that has a null preceding chunk pointer value isselected. Per operation 705, the File ID corresponding with (e.g.,included in) that record is identified.

Per operation 706, a pointer to the current stored chunk (e.g., thechunk wherein the null value was found) is written into the filemetadata for the identified file. This serves to represent that thiscurrent stored chunk is the first (e.g., initial) chunk in theidentified file. Per operation 707, the record containing that nullvalue is marked as recovered. This may be done, for example, byindicating that the record is recovered in a recovery history table(e.g., recovery history table 801 of FIG. 8). Another method of markinga record as recovered includes, in some embodiments, moving a recoverypointer (or other marker) down to the most recently recovered record foreach File ID in each chunk metadata (or a copy of each chunk metadataloaded into memory during a recovery operation). During further reads ofthe chunk metadata, this recovery pointer then serves to indicate whichrecords are recovered for a particular File ID (e.g., the records at orabove (older than) the currently marked record) and which records arenot recovered for the particular File ID (e.g., the records below thecurrently marked record).

Next, per operation 708, a determination is made as to whether thefollowing chunk pointer for this (now recovered) record is null. If not,then, per operation 709, the following chunk pointer for the record isidentified. Per operation 710, the chunk pointed to by that pointer isselected. Per operation 711, the chunk metadata embedded with thatpointed to chunk is read. Per operation 712, the oldest unrecoveredrecord with the identified File ID (e.g., the File ID identified inoperation 705) in this chunk metadata is selected. It should be notedthat this record may, in many instances, be different from the oldestunrecovered record per se in the chunk metadata (e.g., when the oldestrecord per se is for a different file). Next, per operation 706, apointer to the current stored chunk (e.g., the chunk selected inoperation 712) is written to the file metadata for the identified file.This serves to represent that this chunk is the next chunk in theidentified file. In some embodiments, this new pointer is written to alocation immediately following the pointer to the preceding chunk sothat the chunks may be read sequentially during future accesses of theidentified file, which may occur, for example, via method 600 of FIG. 6.Per operation 707, the record is marked as recovered.

The method then proceeds back to operation 708, wherein a newdetermination is made as to whether the following chunk pointer for thecurrent (now recovered) record is null. The method loops throughoperations 706 to 712 until a record with a null pointer value for thefollowing chunk is found. This serves to represent that this currentstored chunk is the last (e.g., final) chunk in the identified file.Once this occurs, the method proceeds to operation 713, wherein theidentified file is marked as recovered. This may be done, for example,by indicating that the identified file is recovered in a recoveryhistory table (e.g., recovery history table 801 of FIG. 8). Peroperation 714, a determination is made as to whether there areadditional unrecovered files (e.g., files in storage media for which thefile metadata has not been recovered and re-added to the metadatastructure). If there are additional unrecovered files, then the methodreturns to operation 701 and loops through operations 701-714 asappropriate. If there are no additional unrecovered files, then, peroperation 715, the method ends.

An example use of operations of the method 700 is now described withreference to the files shown in FIG. 3. In this example, the filemetadata 311 for the Files A, B, and C is lost from a metadata structureof storage media in which the files are stored. This loss is detected,and a recovery operation is begun by the system. As part of the recoveryoperation, the embedded metadata 331 of stored chunk 321 (“Chunk 1”) isread. The first record in embedded metadata 331 is selected as theoldest unrecovered record with a null value for the preceding chunkpointer. The file ID (“File A”) is identified. A pointer to Chunk 1 iswritten to a new version of the file metadata 311 for File A. The recordis then marked as recovered. Because the following chunk pointer for therecord identifies stored chunk 322 (“Chunk 2”), that chunk is selected,the embedded metadata 332 is read, and the first record in embeddedmetadata 332 is selected as the oldest unrecovered record with the FileA identifier. A pointer to Chunk 2 is written to the new version of thefile metadata 311 for File A. The record is then marked as recovered.

Next, because the following chunk pointer for the record identifiesstored chunk 323 (“Chunk 3”), that chunk is selected, the embeddedmetadata 333 is read, and the first record in embedded metadata 333 isselected as the oldest unrecovered record with the File A identifier. Apointer to the Chunk 3 is written to the new version of the filemetadata 311 for File A. The record is then marked as recovered. Becausethe following chunk pointer for the record is null, File A is marked asrecovered.

Having recovered File A, the system proceeds to select Chunk 1 again.Because there are no longer any unrecovered records with null precedingchunk pointers in the embedded metadata 331 of Chunk 1, the systemproceeds to select Chunk 2. The second record in embedded metadata 332is selected as the oldest unrecovered record with a null preceding chunkpointer. The file ID (“File B”) is identified for that record. Becausethe preceding chunk pointer for that record is null, a pointer to Chunk2 is written to a new version of the file metadata 311 for File B. Therecord is then marked as recovered. Because the following chunk pointerfor the record identifies Chunk 1, that chunk is selected, the embeddedmetadata 331 is read, and the second record in embedded metadata 331 isselected as the oldest unrecovered record with the File B identifier. Apointer to the Chunk 1 is written to the new version of the filemetadata 311 for File B. The record is then marked as recovered.

Because the following chunk pointer for the record identifies Chunk 1,that chunk is selected again. The embedded metadata 331 is read, and thethird record in embedded metadata 331 is selected as the oldestunrecovered record with the File B identifier (because the second recordwas already recovered). A pointer to Chunk 1 is written to the newversion of the file metadata 311 for File B. The record is then markedas recovered.

Next, because the following chunk pointer for the record identifiesstored chunk 324 (“Chunk 4”), that chunk is selected, the embeddedmetadata 334 is read, and the first record in embedded metadata 334 isselected as the oldest unrecovered record with the File B identifier. Apointer to Chunk 4 is written to the new version of the file metadata311 for File B. The record is then marked as recovered. Because thefollowing chunk pointer for the record is null, File B is marked asrecovered.

Having recovered Files A and B, the system proceeds to select Chunk 1again. Because there are no longer any unrecovered records with nullpreceding chunk pointers in the embedded metadata 331 of Chunk 1, thesystem proceeds to select Chunk 2. Next, because there are no longer anyunrecovered records with null preceding chunk pointers in the embeddedmetadata 332 of Chunk 2, the system proceeds to select Chunk 3. Theembedded metadata 333 of Chunk 3 is read. The second record in embeddedmetadata 333 is selected as the oldest unrecovered record with a nullpreceding chunk pointer. The file ID (“File C”) is identified for thatrecord. Because the preceding chunk pointer for that record is null, apointer to Chunk 3 is written to a new version of the file metadata 311for File C. The record is then marked as recovered.

Because the following chunk pointer for the record identifies Chunk 2,that chunk is selected, the embedded metadata 332 is read, and the thirdrecord in embedded metadata 332 is selected as the oldest unrecoveredrecord with the File C identifier. A pointer to Chunk 2 is written tothe new version of the file metadata 311 for File C. The record is thenmarked as recovered. Because the following chunk pointer for the recordis null, File C is marked as recovered. The file metadata 311 for allthree of the Files A, B, and C having been recovered, the system endsthe recovery operation.

While example embodiments of the methods have been provided herein, itis contemplated that many variations on these methods may occur. Forexample, in some embodiments, each record in metadata embedded with aparticular chunk may include a single pointer to only one other chunk.This other chunk may be a chunk that is next to (e.g., either precedingor following) the particular chunk in the file to which that recordrelates. Such embodiments (“single-pointer embodiments”) may allow afile to be stored with less chunk metadata than embodiments includingpointers to both chunks (e.g., the one preceding and the one following)that are next to the particular chunk in the relevant file. Suchembodiments may include a first (e.g., initial) chunk indicator and alast chunk indicator for each stored file. These indicators may take theform of indicator bits, null values, or other information within thechunk metadata.

There are at least two examples of these single-pointer embodiments. Ina first example, each record includes a pointer to a preceding chunk anddoes not include a pointer to a following chunk. In this example, arecord associated with an initial chunk in a file may include a nullvalue for the preceding chunk pointer portion of the embedded metadata.Correspondingly, a record associated with a last chunk in that file mayinclude some other indicator (e.g., a known, last chunk identifiervalue) that serves to indicate the last chunk.

To recover file metadata using this first example single-pointerembodiment, the records having a particular file ID may be checked untila record with the last chunk indicator is identified. A pointer to thechunk associated with this record is included in the file metadata forthis file ID as the last pointer in a (to be created) sequence ofpointers. The file ID may then be traced backwards through the records(e.g., via the preceding chunk pointers) until the record with the nullvalue is found. Traced through records are marked as recovered andpointers to the chunks associated with these records are added to thefront of the sequence of pointers. Because the file metadata is beingrecovered backwards, the newest record with the selected file ID (ratherthan the oldest) is selected when tracing through embedded metadata fora given chunk. Once the record with the null value is found, therecovery of the file metadata for that file ID is complete, and a newfile ID is selected. This repeats until the file metadata for all of therelevant files are recovered.

In a second example of these single pointer embodiments, each recordincludes a pointer to a following chunk and does not include a pointerto a preceding chunk. In this example, a record associated with a lastchunk in a file may include a null value for the following chunk pointerportion of the embedded metadata. Correspondingly, a record associatedwith an initial chunk in that file may include some other indicator(e.g., a known, first chunk identifier value) that serves to indicatethe initial chunk.

To recover file metadata using this second example single-pointerembodiment, the records having a particular file ID may be checked untila record with the initial chunk indicator is identified. A pointer tothe chunk associated with this record is included in the file metadatafor this file ID as the first pointer in a (to be created) sequence ofpointers. The file ID may then be traced forwards through the records(e.g., via the following chunk pointers) until the record with the nullvalue is found. Traced through records are marked as recovered (with theoldest unrecovered records with the file ID being selected for use) andpointers to the chunks associated with these records are added to theend of the sequence of pointers. Once the record with the null value isfound, the recovery of the file metadata for that file ID is complete,and a new file ID is selected. This repeats until the file metadata forall of the relevant files are recovered.

Referring now to FIG. 8, shown is a block diagram of an example recoveryhistory table 801, in accordance with embodiments of the presentdisclosure. In some embodiments, recovery history table 801 is used totrack the progress of a file metadata recovery operation. In doing so,the table 801 may serve at least two purposes. First, the table 801allows a recovery operation to be stopped (either intentionally orunintentionally) and restarted at a later time without having to startthe entire operation over again from the beginning. Second, the tablemay also include a listing of recovered records 813 and recovered files812.

In some embodiments, table 801 can serve as the location as the placewhere records and files are marked as recovered. For example, the recordidentifier (record ID) for a newly recovered record may be added to therecovered records listing 813. Then during the course of the recoveryoperation, the record ID of selected record may be checked against thelist in order to determine whether it has been recovered or not.Likewise, using the recovered files listing 812, the identity of thefiles that have been recovered can be determined. In addition, the table801 may include a progress bar 811 that indicates to a user how muchlonger the recovery operation will take to complete.

Some embodiments of the present disclosure may offer various technicalcomputing advantages over the prior art. These computing advantagesaddress problems arising in the realm of computer storage systems andthe associated problems of computer performance and reliability thatoccur when metadata is lost or corrupted. Implementation of embodimentsof the method 700, for example, can result in improved systemperformance and technical computing advantages. Embodiments hereinrecognize that using pointers in metadata embedded with data chunks canhave significant advantages (e.g., in terms of computing resources) infile metadata recovery

Referring now to FIG. 9, shown is a high-level block diagram of anexample computer system (i.e., computer) 901 that may be used inimplementing one or more of the methods or modules, and any relatedfunctions or operations, described herein (e.g., using one or moreprocessor circuits or computer processors of the computer), inaccordance with embodiments of the present disclosure. In someembodiments, the major components of the computer system 901 maycomprise one or more CPUs 902, a memory subsystem 904, a terminalinterface 912, a storage interface 914, an I/O (Input/Output) deviceinterface 916, and a network interface 919, all of which may becommunicatively coupled, directly or indirectly, for inter-componentcommunication via a memory bus 903, an I/O bus 909, and an I/O businterface unit 910.

The computer system 901 may contain one or more general-purposeprogrammable central processing units (CPUs) 902A, 902B, 902C, and 902D,herein generically referred to as the processor 902. In someembodiments, the computer system 901 may contain multiple processorstypical of a relatively large system; however, in other embodiments thecomputer system 901 may alternatively be a single CPU system. Each CPU902 may execute instructions stored in the memory subsystem 904 and maycomprise one or more levels of on-board cache.

In some embodiments, the memory subsystem 904 may comprise arandom-access semiconductor memory, storage device, or storage medium(either volatile or non-volatile) for storing data and programs. In someembodiments, the memory subsystem 904 may represent the entire virtualmemory of the computer system 901, and may also include the virtualmemory of other computer systems coupled to the computer system 901 orconnected via a network. The memory subsystem 904 may be conceptually asingle monolithic entity, but, in some embodiments, the memory subsystem904 may be a more complex arrangement, such as a hierarchy of caches andother memory devices. For example, memory may exist in multiple levelsof caches, and these caches may be further divided by function, so thatone cache holds instructions while another holds non-instruction data,which is used by the processor or processors. Memory may be furtherdistributed and associated with different CPUs or sets of CPUs, as isknown in any of various so-called non-uniform memory access (NUMA)computer architectures. In some embodiments, the main memory or memorysubsystem 904 may contain elements for control and flow of memory usedby the Processor 902. This may include a memory controller 905.

Although the memory bus 903 is shown in FIG. 9 as a single bus structureproviding a direct communication path among the CPUs 902, the memorysubsystem 904, and the I/O bus interface 910, the memory bus 903 may, insome embodiments, comprise multiple different buses or communicationpaths, which may be arranged in any of various forms, such aspoint-to-point links in hierarchical, star or web configurations,multiple hierarchical buses, parallel and redundant paths, or any otherappropriate type of configuration. Furthermore, while the I/O businterface 910 and the I/O bus 909 are shown as single respective units,the computer system 901 may, in some embodiments, contain multiple I/Obus interface units 910, multiple I/O buses 909, or both. Further, whilemultiple I/O interface units are shown, which separate the I/O bus 909from various communications paths running to the various I/O devices, inother embodiments some or all of the I/O devices may be connecteddirectly to one or more system I/O buses.

In some embodiments, the computer system 901 may be a multi-usermainframe computer system, a single-user system, or a server computer orsimilar device that has little or no direct user interface, but receivesrequests from other computer systems (clients). Further, in someembodiments, the computer system 901 may be implemented as a desktopcomputer, portable computer, laptop or notebook computer, tabletcomputer, pocket computer, telephone, smart phone, mobile device, or anyother appropriate type of electronic device.

It is noted that FIG. 9 is intended to depict the representative majorcomponents of an exemplary computer system 901. In some embodiments,however, individual components may have greater or lesser complexitythan as represented in FIG. 9, components other than or in addition tothose shown in FIG. 9 may be present, and the number, type, andconfiguration of such components may vary.

As discussed in more detail herein, it is contemplated that some or allof the operations of some of the embodiments of methods described hereinmay be performed in alternative orders or may not be performed at all;furthermore, multiple operations may occur at the same time or as aninternal part of a larger process.

The present invention may be a system, a method, and/or a computerprogram product. The computer program product may include a computerreadable storage medium (or media) having computer readable programinstructions thereon for causing a processor to carry out aspects of thepresent invention.

The computer readable storage medium can be a tangible device that canretain and store instructions for use by an instruction executiondevice. The computer readable storage medium may be, for example, but isnot limited to, an electronic storage device, a magnetic storage device,an optical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. A non-exhaustive list of more specific examples of thecomputer readable storage medium includes the following: a portablecomputer diskette, a hard disk, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), a static random access memory (SRAM), a portablecompact disc read-only memory (CD-ROM), a digital versatile disk (DVD),a memory stick, a floppy disk, a mechanically encoded device such aspunch-cards or raised structures in a groove having instructionsrecorded thereon, and any suitable combination of the foregoing. Acomputer readable storage medium, as used herein, is not to be construedas being transitory signals per se, such as radio waves or other freelypropagating electromagnetic waves, electromagnetic waves propagatingthrough a waveguide or other transmission media (e.g., light pulsespassing through a fiber-optic cable), or electrical signals transmittedthrough a wire.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network, for example, the Internet, a local area network, awide area network and/or a wireless network. The network may comprisecopper transmission cables, optical transmission fibers, wirelesstransmission, routers, firewalls, switches, gateway computers, and/oredge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device.

Computer readable program instructions for carrying out operations ofthe present invention may be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, or either source code or object code written in anycombination of one or more programming languages, including an objectoriented programming language such as Smalltalk, C++ or the like, andconventional procedural programming languages, such as the “C”programming language or similar programming languages. The computerreadable program instructions may execute entirely on the user'scomputer, partly on the user's computer, as a stand-alone softwarepackage, partly on the user's computer and partly on a remote computeror entirely on the remote computer or server. In the latter scenario,the remote computer may be connected to the user's computer through anytype of network, including a local area network (LAN) or a wide areanetwork (WAN), or the connection may be made to an external computer(for example, through the Internet using an Internet Service Provider).In some embodiments, electronic circuitry including, for example,programmable logic circuitry, field-programmable gate arrays (FPGA), orprogrammable logic arrays (PLA) may execute the computer readableprogram instructions by utilizing state information of the computerreadable program instructions to personalize the electronic circuitry,in order to perform aspects of the present invention.

Aspects of the present invention are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer readable program instructions.

These computer readable program instructions may be provided to aprocessor of a general purpose computer, special purpose computer, orother programmable data processing apparatus to produce a machine, suchthat the instructions, which execute via the processor of the computeror other programmable data processing apparatus, create means forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks. These computer readable program instructionsmay also be stored in a computer readable storage medium that can directa computer, a programmable data processing apparatus, and/or otherdevices to function in a particular manner, such that the computerreadable storage medium having instructions stored therein comprises anarticle of manufacture including instructions which implement aspects ofthe function/act specified in the flowchart and/or block diagram blockor blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof instructions, which comprises one or more executable instructions forimplementing the specified logical function(s). In some alternativeimplementations, the functions noted in the block may occur out of theorder noted in the figures. For example, two blocks shown in successionmay, in fact, be executed substantially concurrently, or the blocks maysometimes be executed in the reverse order, depending upon thefunctionality involved. It will also be noted that each block of theblock diagrams and/or flowchart illustration, and combinations of blocksin the block diagrams and/or flowchart illustration, can be implementedby special purpose hardware-based systems that perform the specifiedfunctions or acts or carry out combinations of special purpose hardwareand computer instructions.

As used herein, the term “each” does not necessarily equate to the term“all” as the term “all” is used colloquially. For example, the followingtwo phrases have different meanings: “a car having a plurality of tires,each tire of the plurality of tires being fully inflated” and “a carthat has all of its tires fully inflated”. The former phrase wouldencompass a car with three fully-inflated tires (the plurality of tires)and one flat tire (not included in the plurality of tires). The latterphrase would not encompass such a car (because not all of the car'stires are fully inflated). Likewise, the phrase “a computer having a setof files, each file of the set of files being read-only” would encompassa computer having two files, one of which is read-only (and belongs tothe set of files) and one of which is not read-only (and does not belongto the set of files).

The descriptions of the various embodiments of the present disclosurehave been presented for purposes of illustration, but are not intendedto be exhaustive or limited to the embodiments disclosed. Manymodifications and variations will be apparent to those of ordinary skillin the art without departing from the scope and spirit of the describedembodiments. The terminology used herein was chosen to best explain theprinciples of the embodiments, the practical application or technicalimprovement over technologies found in the marketplace, or to enableothers of ordinary skill in the art to understand the embodimentsdisclosed herein.

Although the present invention has been described in terms of specificembodiments, it is anticipated that alterations and modification thereofwill become apparent to the skilled in the art. Therefore, it isintended that the following claims be interpreted as covering all suchalterations and modifications as fall within the true spirit and scopeof the invention.

1. A method comprising: storing a data stream in storage media, whereinthe storing the data stream comprises: dividing a file within the datastream into a plurality of chunks, the plurality of chunks including afirst chunk that is an initial chunk in the file that is followed, inthe file, by a second chunk that is followed, in the file, by a thirdchunk that is followed, in the file, by a fourth chunk that is followed,in the file, by a fifth chunk that is a final chunk in the file;determining that the first chunk matches a first existing chunk storedin the storage media; creating, in response to the determining that thefirst chunk matches the first existing stored chunk, a first pointer tothe first existing stored chunk in file metadata for the file; creatinga first recovery bit and a null value in a first record in chunkmetadata embedded with the first existing stored chunk; determining thatthe second chunk does not match any existing chunk in the storage media;storing, in response to the determining that the second chunk does notmatch any existing stored chunks, the second chunk in the storage mediaas a second stored chunk; creating a second pointer to the second storedchunk in the file metadata for the file; creating a second recovery bitand a third pointer to the first existing stored chunk in a secondrecord in chunk metadata embedded with the second stored chunk;determining that the third chunk matches a third existing chunk storedin the storage media; creating, in response to the determining that thethird chunk matches the third existing stored chunk, a fourth pointer tothe third existing stored chunk in file metadata for the file; creatinga third recovery bit and a fifth pointer to the second stored chunk in athird record in chunk metadata embedded with the third existing storedchunk; determining that the fourth chunk matches the second storedchunk; creating, in response to the determining that the fourth chunkmatches the second stored chunk, a sixth pointer to the second storedchunk in file metadata for the file; creating a fourth recovery bit anda seventh pointer to the third existing stored chunk in a fourth recordin chunk metadata embedded with the second stored chunk; determiningthat the fifth chunk matches a fourth existing chunk stored in thestorage media; creating, in response to the determining that the fifthchunk matches the fourth existing stored chunk, an eighth pointer to thefourth existing stored chunk in file metadata for the file; and creatinga fifth recovery bit, a final chunk indicator value, and a ninth pointerto the second stored chunk in a fifth record in chunk metadata embeddedwith the fourth existing stored chunk; detecting that the file metadatais lost or corrupted; and performing, in response to the detecting, arecovery operation, wherein the recovery operation comprises: readingthe fifth record in chunk metadata embedded with the fourth existingstored chunk; determining, based on reading the final chunk indicatorvalue and the ninth pointer in the fifth record in chunk metadataembedded with the fourth existing stored chunk, that the fifth chunkdoes not have any following chunks in the file and that the fifth chunkfollows the fourth chunk in the file; setting the fifth recovery bit;reading the fourth record in chunk metadata embedded with the secondstored chunk; determining, based on reading the seventh pointer in thefourth record in chunk metadata embedded with the second stored chunk,that the fourth chunk follows the third chunk in the file; setting thefourth recovery bit; reading the third record in chunk metadata embeddedwith the third existing stored chunk; determining, based on reading thefifth pointer in the third record in chunk metadata embedded with thethird existing stored chunk, that the third chunk follows the secondchunk in the file; setting the third recovery bit; reading the secondrecord in chunk metadata embedded with the second stored chunk;determining, based on reading the third pointer in the second record inchunk metadata embedded with the second stored chunk, that the secondchunk follows the first chunk in the file; setting the second recoverybit; reading the first record in chunk metadata embedded with the firstexisting stored chunk; determining, based on reading the null value inthe first record in the chunk metadata embedded with the first existingstored chunk, that the first chunk does not have any preceding chunks inthe file; setting the first recovery bit; and creating a recoveredversion of the file metadata, wherein, based on the determining that thetarget chunk does not have any preceding chunks in the file, thedetermining that the second chunk follows the first chunk in the file,the determining that the third chunk follows the second chunk in thefile, the determining that the fourth chunk follows the third chunk inthe file, the determining that the fifth chunk follows the fourth chunkin the file, and the determining that the fifth chunk does not have anyfollowing chunks in the file, the recovered version of the file metadataindicates that the first chunk is the initial chunk in the file that isfollowed, in the file, by the second chunk that is followed, in thefile, by the third chunk that is followed, in the file, by the fourthchunk that is followed, in the file, by the fifth chunk that is thefinal chunk in the file.